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INTER-FRAME VIDEO TECHNIQUES FOR BEHAVIORAL ANALYSIS 
OF LABORATORY ANIMALS 

FIELD OF THE INVENTION: 

The present invention is generally directed to monitoring animal behavior and, more 
particularly, to monitoring animal behavior to determine the presence or lack thereof of a fear 
reaction. 

5 

BACKGROUND 

The correlation of neural signals ("brainwaves") or neurological impulses in test 
animals to external and other stimuli is an important part of biological study. One way to 
correlate animal brainwaves to external stimuli is to track a test animal's position over time, 

10 in terms of its center of mass ("centroid"). The change of position of the test animal can be 
the result of external stimuli. The change of position of the test animal is also manifest as 
changes in brainwaves. The brainwaves can be quantified through use of a device that can 
measure electrical signals produced by individual neurons by way of one or more electrodes 
implanted in the brain or other neural materials, such as the spinal column. The change of the 

1 5 centroid over time can also be quantified. Therefore, reactions by the test animal to external 
stimuli can be quantified by correlating any changes in the centroid to changes of brainwave 
activity. 

Typically, as the animal moves, perhaps as the result of external stimuli, the animal's 
image can be recorded many times per second by a video camera. The brainwaves of the test 

2 o animal are also recorded by a recording device, such as a multi-channel acquisition processor 
(MAP), wherein the recordings are also time-stamped. Generally, a MAP can be defined as a 
device that is capable of recording and time-stamping at least two separate data streams at 
the same time. The position of the test animal, as indicated by its centroid, is calculated by a 
tracking device. The calculated position of the centroid is also then recorded and 

25 time-stamped by the MAP. After both the brainwaves and the centroid positions have been 
time-stamped and recorded by the MAP, researchers can then correlate the macroscopic 
behavior of the animal with its concurrent neural activity. 

Within the tracking device, the position of the test animal is determined by the 
calculation of the centroid from video frame to video frame. After each video frame is 

1 
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acquired by the camera, the frame is then sent to the tracking device. The tracking device 
performs various operations upon each video frame, such as deleting the image of the 
animal's tail and the image of any wires attached to the animal's skull, increasing the 
contrast between the test animal and the background, and so on. After these operations are 
5 performed, the centroid of the pixels corresponding to the test animal for each video frame is 
calculated. As discussed above, the centroid can be generally defined as the average 
"middle" of the animal. Typically, the centroid is plotted in an "X-Y axis'Vtwo dimensional 
coordinate plane. The test animal's position can be precisely mapped to brainwave 
measurements. Therefore, reactions by the animal, such as muscular movements as a result 

10 of outside stimuli, can be mapped to brainwaves. 

However, certain reactions of note of the test animal do not lend themselves to be 
readily determined by centroid calculation. There are some reactions to external stimuli that 
can not be readily ascertained as a function of a change of centroid. One such reaction of the 
test animal is the "fear" reaction. Typically, in a fear reaction, animals "freeze" having less 

15 movement than they would if they were merely at rest, sometimes not even breathing until 
the fear reaction abates. In a fear reaction, the brain of the test animal generates brainwaves, 
which can be recorded by the MAP. For a researcher, discerning the brainwaves for a fear 
reaction from the brainwaves from an animal not in fear but "at rest" is a difficult task. 
Generally, this is because a primary indicator of animal behavior, the position of the centroid 

20 from video frame to video frame, may not change. 

Therefore, there is a need for a method and a system to discern whether an animal is in 
a fear reaction that overcomes at least some of the problems associated with conventional 
fear reaction discernment systems or methods. 



2 



DOCKET NO. PLEX 2781000 



PATENT APPLICATION 



SUMMARY OF THE INVENTION 

The present invention provides for determining the extent of movement of an object 
appearing within two or more images. Each point in the images is classified as either on- 
object or off-object to create classified images. One classified image is compared to at least 
5 one other classified image to determine those areas for which the classification differs. At 
least one neurological impulse is measured. The determination of those areas for which the 
classification differs to the measurement of the at least one neurological impulse is 
correlated. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present invention, and the advantages 
thereof, reference is now made to the following description taken in conjunction with the 
accompanying drawings, in which: 
5 FIGURE 1 illustrates a prior art system for correlating centroid position to recorded 

brainwaves; 

FIGURE 2 illustrates a video tracker for determining a change of corresponding 
pixels from video frame to video frame; 

FIGURE 3A illustrates a change of extent and centroid position of a test animal 
l o between a first video frame and a second video frame when the animal is not in a fear mode; 

FIGURE 3B illustrates the extent and centroid of a test animal in a fear mode 
remaining relatively stationary between a first video frame and a second video frame; 

FIGURE 3C illustrates the extent and centroid of a test animal in a substantially 
stationary grooming mode, wherein the test animal extent is significantly greater in the 
15 second video frame than the first video frame; 

FIGURE 4 illustrates an XOR processor; and 

FIGURE 5 illustrates a method for determining a degree of similarity of pixels 
occupied by an object in a first video frame and in a second video frame. 
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DETAILED DESCRIPTION 

In the following discussion, numerous specific details are set forth to provide a 
thorough understanding of the present invention. However, it will be understood that the 
present invention can be practiced by those skilled in the art following review of this 
5 description, without such specific details. In other instances, well-known elements have 
been illustrated in schematic or block diagram form in order not to obscure the present 
invention in unnecessary detail. Additionally, for the most part, details concerning feedback 
systems and the like have been omitted inasmuch as such details are not considered 
necessary to obtain a complete understanding of the present invention, and are considered to 
10 be within the skills of persons of ordinary skill in the relevant art. 

It is further noted that, unless indicated otherwise, all functions described herein are 
performed by a processor such as a computer or electronic data processor in accordance with 
code such as computer program code, software, and/or integrated circuits that are coded to 
perform such functions. 

15 Turning now to FIGURE 1, illustrated is a prior art system 100 for measuring the 

change of a centroid position of a test animal from a first video frame to a second video 
frame. In the system 1 00, a test animal 1 85 is allowed to move within a receptacle 1 80. The 
system 100 tracks the change of a centroid 190 from video frame to video frame, and 
correlates these changes to measured brainwaves as a function of time. From these 

2 0 brainwave measurements and any centroid change in position, the correlation of brainwaves 
to behavior of the test animal can be deduced. 

In the system 100, the test animal 185 is illuminated by two light sources 151, 152. 
The brain and nervous system of the test animal 185 is coupled to a brainwave acquisition 
processor and concentrator 160 through electrode wires 170 or other suitable means. The 

25 concentrator conveys the measured brainwaves of the test animal 185 to the MAP 123. 
Within the MAP 123, the measured brainwaves are also time-stamped. 

The test animal 185 is also photographed at regular intervals, such as 30 times per 
second, by a camera 150. The resulting image ("video frame") is sent to the video tracker 
125. Within the video tracker 125, digital signal processing (DSP) is performed upon each 

30 picture element or "pixel" of the image. One of the DSP actions performed by the video 
tracker 125 is that of determining the centroid 190 of the test animal 185 for each video 
frame. This information is then conveyed to the MAP 123. The MAP 123 then time stamps 
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the position of the centroid for each video frame. The MAP 123 then correlates the 
timestamp for the measured brainwaves to the timestamp for the centroid for each video 
frame. This correlation is then sent to a computing device 120, such as a personal computer 
(PC) for further analysis. Video frames, including their corresponding centroids, may be sent 
5 to the PC 120 for display. 

The PC 120 acts as a server for a display 1 10. The PC 120 can also be used to filter 
and perform further digital signal processing upon the measured brainwaves time-stamped by 
the MAP 1 23. The display 1 1 0 can show a video frame, with the calculated centroid, and the 
change of centroid from video frame to video frame. The display 1 1 0 can also show various 
10 brainwaves, both in the original form of graphed raw data and processed data. The 
processing of the brainwaves can be performed as a function of parameters input by a 
researcher. 

Turning now to FIGURE 2, illustrated is a video tracker 200. Generally, the video 
tracker 200 is configured to exploit a central fact of fear reactions of animals, such as test 

1 5 animals. Typically, animals "freeze up" when undergoing a fear reaction. The fear reaction 
leads to less movement between frames, such as evinced by a cessation of breathing by the 
test animal, than would otherwise be present in an animal not in fear mode. While the lack 
of a change in a centroid position from video frame to video frame is a necessary condition to 
indicate the presence of a fear mode in a test animal, it is not a sufficient condition, as the 

2 o test animal could be at rest, grooming, or otherwise occupied rather than exhibiting a true 
fear reaction. 

Generally, in the video tracker system 200, a first video frame is compared to a 
second video frame. First, each pixel of each frame is determined to lie either on the animal 
("animal pixel") or off the animal ("non-animal pixel"). Then the number of pixels which 

2 5 changed state (from animal- to non-animal or non-animal to animal) from the first to the 
second video frame is determined. This number can then be normalized by dividing it by the 
total number of animal pixels in the second video frame, for example. Dividing the number 
of changed pixels by the total number of animal pixels in the second frame generates a 
normalized value with a range from 0.0 to 2.0 in the case where the animal extent (number of 

30 pixels occupied) does not change. Generally, the lower the normalized or non-normalized 
value, the more likely that the animal is in a fear reaction. The normalized or non- 
normalized value can be output by a video tracker 220 and into a MAP 295 for time- 
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stamping and correlating to measured brainwaves. This normalized or non-normalized value 
may be referred to as the "inter-frame motion measure" or simply "motion measure". 

Within a video tracker 220, a video camera 205 is coupled to a video capture card 
210. Each video frame is temporarily stored within the video capture card 210. Then, the 
5 video frame is conveyed to the video tracker 220. Alternately, a digital camera with a digital 
interface such as Institute of Electrical and Electronics Engineers (IEEE) 1394 camera, can 
be used, which obviates the need for a separate capture card 210. 

Within the video tracker 220, discrimination tasks are performed. Generally, the 
discrimination tasks are image processing operations that help determine where the test 

10 animal is within the digitized photograph. When the discrimination task is complete, the 
status of each pixel of each video frame has been determined to be either on-animal or off- 
animal. For each pair of processed video frames, the video tracker 220 counts the number of 
pixels that have changed status, i.e. changed from on-animal to off-animal or from off-animal 
to on-animal. The video tracker 220 optionally normalizes the result by dividing by the 

15 number of on-animal pixels in the second frame, for example. This generates a quantitative 
normalized value for the likelihood that the test animal is in a fear reaction, with lower 
values indicating a higher likelihood of a fear reaction. 

Within the video tracker 220, the first processing typically performed is a threshold 
operation 230. Within the threshold operation 230, the light intensity value of each pixel is 

20 compared to a configurable threshold value. The threshold operation 230 typically sets 
recorded pixels with light intensity values less than the configurable threshold value to zero, 
and all other pixel values to the maximum allowed pixel value, for a video frame. 
Alternately, pixel values equal to or greater than the threshold are set to a value of one. The 
thresholded pixel values are then sent to the inverter 240. If the test animal is dark and the 

25 background light, an inversion operation can be applied which sets zero-value pixels to the 
largest value, and the largest value pixels to zero. The application of the inverter 240 can 
insure that animal pixels are non-zero, even when the intensity of the reflected light from the 
test animal is lower than the intensity of the reflected light from the background. 

After any inversion has been performed in the inverter 240, any necessary detail 

30 removal is performed in the detail remover 250. In the detail remover 250, the image of such 
things as the tail of the test animal is removed so as to not impair the accuracy of the centroid 
calculation as an estimate of the center of mass of the test animal. Furthermore, the tail may 
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be deleted from consideration from the determination of the normalized change of pixel 
status. Similarly, any wires coupled to the head of the test animal can skew the change of 
pixel status calculation, as the wires could be moving, even if the test animal is not moving 
due to a fear reaction. Therefore, the wires are deleted from consideration prior to the 
5 calculation of the centroid or of the inter-frame motion measure calculation. 

From the detail remover 250, the processed digitized pixilated values are sent to an 
optional masking operation 260. Generally, the mask 260 is used to delete an area or areas 
within the video frame which have the same or a similar intensity of reflected light as the test 
animal, and which would otherwise erroneously affect the calculation of the centroid or the 

10 inter- frame motion detection calculation. The appropriate mask or masks are stored in a 
mask image store 270. The processed image is then stored in memory 275, for use in 
comparison by the XOR processor 285. In an alternative embodiment, the processed video 
frames are stored in memory of the XOR processor 285, itself. 

After the mask image from mask image store 270 is applied in the mask operation 

1 5 260, the processed and discriminated image of the test animal can be displayed on a display 
297. The centroid of the test animal for a video frame can be calculated in the centroid 
calculator 280. The centroid calculator 280 can output the centroid calculation as coordinate 
points in a Cartesian coordinate system, although other coordinate systems are within the 
scope of the present invention. 

20 The change in pixel status (on-animal or off-animal) between video frames can be 

calculated in an XOR processor 285. Determining any change in the pixel by pixel status 
enables the determination of the degree of similarity of the precise extent occupied by a test 
animal between video frames. A low normalized or non-normalized value of this "motion 
measure", indicating high similarity, can be highly indicative of a fear reaction on the part of 

25 the test animal. 

In the video tracker 220, the inter-frame motion measure can be calculated by 
exclusively ORing (XORing) within the XOR processor 285 each pixel in the current video 
frame after discrimination with its corresponding pixel in the previous video frame after 
discrimination. The discrimination video frame can be generally defined as each video frame 

30 after the discrimination operations have been applied. 

As is understood by those of skill in the art, an XOR function returns a "true" value, a 
logical value of one, if both the input values to the XOR logic are different, and a false value, 



DOCKET NO. PLEX 2781000 



PATENT APPLICATION 



a logical value of zero, if both of the input values are the same. More particularly, in the 
XOR processor 285, if the same pixel of two separate video frames have the same value (that 
is, in both frames, both of the corresponding pixels are either on-animal or both are off- 
animal pixels), the test animal has not moved in or out of the area represented by that pixel 
5 (that is, there has been "no movement" for that pixel). Therefore, the XOR processor 285 
generates a false value for that pixel. However, if the value of the corresponding pixels of 
both video frames is different, then the test animal has moved in or out of that pixel. This is 
expressed within the XOR processor 285 if there is a change between the two corresponding 
pixels (that is, the test animal has moved in or out of the pixel) and a logical true value is 

1 0 generated for that pixel. 

In a further embodiment, the total number of changed pixels, that is, the count of the 
number of true values output by the XOR processor 285, is divided by the total number of 
on-animal pixels in the second/later video frame. This creates a normalized measure of 
animal movement. The normalized level of movement can then be interpreted by researchers 

15 as to whether it indicates a fear or non-fear reaction. 

The centroid information for each video frame is output from the centroid calculator 
280, and well as the motion measure derived from pixel comparisons of video frames by the 
XOR processor 285, and submitted to the MAP 295. This outputted data is useful for 
detecting a fear reaction in a test animal. The centroid results and motion measure results are 

20 time-stamped by the MAP 285 and may therefore be correlated to each other in time by 
processor 296 or other processing equipment. The MAP 295 also timestamps the received 
indicia of brainwave activity of the test animal as received from a brainwave signal 
concentrator 292 coupled to the test animal through electrical wires or other suitable means. 
The MAP 295 also thereby allows correlation in time of centroid, motion measure, and 

25 neural activity and neurological impulses of the test animal. 

Turning now to FIGURE 3 A, illustrated are two video frames which show an animal 
moving between two points in time. The first frame animal extent 308 has a first centroid 
306 (as determined by the centroid calculator 280) and a second frame animal extent 309 has 
a second centroid 307, centroids 306 and 307 corresponding to the position of the test animal 

30 at the times when the frames were captured. There is one area of similarity 330 between the 
first frame animal extent 308 and the second frame animal extent 309. There are also areas 
of difference 3 10 and 320. As will be clear to one of ordinary skill in the art, the result of the 
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pixel-by-pixel XOR operation within the XOR processor 285 are the areas of difference 3 1 0 
and 320. The XOR processor 285 additionally counts the number of pixels within the areas 
of difference 310 and 320 to determine the "motion measure". The XOR processor 285 
optionally normalizes the motion measure by dividing it by the number of animal pixels in 
5 the second frame animal extent 309. A low value of this motion measure (normalized or 
non-normalized) can indicate a fear reaction on the part of the test animal. In the illustration 
of FIG. 3 A, this is a relatively high normalized ratio, probably indicating the lack of a fear 
reaction upon the part of the test animal. 

Turning now to FIGURE 3B, illustrated are two video frames which show an animal 

1 o with minimal movement between two points in time. The first frame animal extent 358 has a 

first centroid 356 (as determined by the centroid calculator 280) and a second frame animal 
extent 359 has a second centroid 357, centroids 356 and 357 corresponding to the position of 
the test animal at the times when the frames were captured. There is one area of similarity 
380 between the first frame animal extent 358 and the second frame animal extent 359. 

1 5 There are also areas of difference 360 and 370. As will be clear to one of ordinary skill in the 
art, the result of the pixel-by-pixel XOR operation within the XOR processor 285 is the areas 
of difference 360 and 370. The XOR processor 285 additionally counts the number of pixels 
within the areas of difference 310 and 320 to determine the "motion measure". The XOR 
processor 285 optionally normalizes the motion measure by dividing it by the number of 

20 animal pixels in the second frame animal extent 309. A low value of this motion measure 
(normalized or non-normalized) can indicate a fear reaction on the part of the test animal. In 
the illustration of FIG. 3B, this is both a very small change in centroid location as well as 
very low inter- frame motion measure, probably indicating a fear reaction upon the part of the 
test animal. 

2 5 Turning now to FIGURE 3C, illustrated are two video frames which show an animal 

moving between two points in time. The first frame animal extent 396 has a first centroid 
39 1 (as determined by the centroid calculator 280) and a second frame animal extent 397 has 
a second centroid 392, centroids 391 and 392 corresponding to the position of the test animal 
at the times when the frames were captured. There is one area of similarity 394 between the 
30 first frame animal extent 396 and the second frame animal extent 397. There is also an area 
of difference 393. As will be clear to one of ordinary skill in the art, the result of the pixel- 
by-pixel XOR operation within the XOR processor 285 is the areas of difference 393. The 
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XOR processor 285 additionally counts the number of pixels within the areas of difference 
310 and 320 to determine the "motion measure". The XOR processor 285 optionally 
normalizes the motion measure by dividing it by the number of animal pixels in the second 
frame animal extent 397. A low value of this motion measure (normalized or non- 
5 normalized) can indicate a fear reaction on the part of the test animal. In the illustration of 
FIG. 3C, the centroid locations 391 and 392 have not changed from the first to the second 
frame, but the inter- frame motion measure remains relatively high. In this case the animal is 
probably not experiencing a fear reaction, but is more likely breathing or grooming in a 
stationary position. 

l o Turning now to FIGURE 4, disclosed is frame comparison logic, an XOR processor 

400. The XOR processor 400 receives data representing pixels for the video frame after the 
masking over a bus 410. The data is then split up into a first memory 430 and a second 
memory 440. In the first and second registers 430, 440 are the processed pixel statuses for 
every pixel of the first and a second video frame. 

15 Each pair of pixel statuses is compared within the XOR 450. For instance, the 

discriminated value/ status for pixel (4,4) of the first video frame is compared to the 
discriminated value/status for pixel (4,4) of the second video frame. If both statuses for the 
two corresponding pixels are the same, then a logical value of false ("0") is output by the 
XOR 450. Otherwise, a logical value of true (" 1 ") is output by the XOR 450. These values 

20 are then input into an aggregator 460. Within the aggregator 460, all of the logical true 
values are counted. Coupled to the aggregator 460 is a normalizer 470. The normalizer 470 
divides the number of "true" values recorded by the aggregator 460 by the total number of 
pixels corresponding to the test animal in the second frame. In an alternate embodiment, the 
normalizer 470 divides the number of "true" values recorded by the aggregator 460 by the 

25 total number of pixels corresponding to the test animal in the first frame. Other 
normalization techniques are possible. This generates a normalized output, the magnitude of 
which is inversely proportional to the likelihood of the test animal is undergoing a fear 
reaction. This normalized output can then be combined with centroid displacement 
information between video frames. The normalized value is transmitted over a bus 420 to 

30 the MAP 295. In an alternative embodiment, the MAP 295 receives the output from the 
XOR 450 without further processing, and the MAP 295 or some other device performs the 
functions of the aggregator 460 and the normalizer 470. 

11 
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Turning now to FIGURE 5, disclosed is a method 500 for determining the normalized 
change of animal pixels from video frame to video frame. This normalized value represents 
the relative fear reaction of a test animal, and is a function of an inter- frame motion detection 
calculation of the XOR logic 400. In step 510, a first video frame is captured of the test 
5 animal. In step 520, the various discrimination techniques are performed upon the first video 
frame leading to each pixel of the frame being given a status of either on-animal or off- 
animal. In the preferred embodiment, on-animal pixels are given a value of ' V representing 
4 true' and off-animal pixels are given a value of '0' representing 'false'. These 
discrimination techniques can include threshold comparison, inversion, detail removal, and 

10 soon. In step 530, a second video frame is captured. In step 540, similar processing is 
performed on the second video frame as is performed on the first video frame. 

In step 550, XOR values are generated for each corresponding pixel on the first and 
second video frames. These values are a logical high if the discriminated pixels change and a 
logical low if the pixels stay the same. In step 560, the number of logical highs is counted 

1 5 within the aggregator, the processor 296, or other processing device. In step 570, the number 
of on-animal pixels in the second video frame is determined. In step 580, the number of 
changed discrimination pixels is divided by the total number of animal pixels within the 
second frame, thereby generating a normalized value which corresponds to an inter-frame 
motion measure. 

20 It is understood that the present invention can take many forms and embodiments. 

Accordingly, several variations can be made in the foregoing without departing from the 
spirit or the scope of the invention. For example, video frame 1 and video frame 2 need not 
be consecutive video frames. As a further example, the techniques presented can be 
extended to consider more than two frames. 

25 Having thus described the present invention by reference to certain of its preferred 

embodiments, it is noted that the embodiments disclosed are illustrative rather than limiting 
in nature and that a wide range of variations, modifications, changes, and substitutions are 
contemplated in the foregoing disclosure and, in some instances, some features of the present 
invention can be employed without a corresponding use of the other features. Many such 

30 variations and modifications can be considered obvious and desirable by those skilled in the 
art based upon a review of the foregoing description of preferred embodiments. 
Accordingly, it is appropriate that the appended claims be construed broadly and in a manner 
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